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Amendments to the Claims: 

This listing of claims will replace all prior versions and listings of claims in the application. The 
present amendment makes no amendment to the claims. 

Listing of Claims: 

Listing of Claims; 

1-40 (Cancelled) 

41 . (Previously Presented) A system for electronically providing performance 
feedback based upon a comparison of an evaluator*s performance in evaluating data items to 
predetermined performance criteria, the system comprising: 

a) a receiving subsystem configured to electronically receive information over a 
distributed computer network related to a first performance level of a first evaluator*s evaluation 
of data items, and a predetermined performance level; 

b) a comparison subsystem configured to electronically compare the first performance 
level and the predetermined performance level; and 

c) a discrepancy subsystem configured to electronically report a discrepancy based on 
predefined criteria between the first performance level and the predetermined performance level. 

42. (Previously Presented) The system of claim 41 wherein the discrepancy 
subsystem is fiirther configured to electronically report the discrepancy to the first evaluator over 
a distributed computer network. 

43. (Previously Presented) The system of claim 41 wherein the discrepancy 
subsystem is fiirther configured to electronically report the discrepancy to a system manager over 
a distributed computer network. 

44. (Previously Presented) A system for electronically providing performance 
feedback based upon a comparison of an evaluator's performance in evaluating data items to 
predetermined performance criteria, the system comprising: 
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a) a receiving subsystem configured to electronically receive information over a 
distributed computer network related to a first validity level of a first evaluator's evaluation of 
data items, and a predetermined validity level; 

b) a comparison subsystem configured to electronically compare the first validity level 
and the predetermined validity level; and 

c) a discrepancy subsystem configured to electronically report a discrepancy based on 
predefined criteria between the first validity level and the predetermined validity level. 

45. (Previously Presented) The system of claim 44 wherein the receiving 
subsystem is fiulher configured to receive an evaluation result for a quality item firom the first 
evaluator over a distributed computer network and a known evaluation result for the quality item. 

46. (Previously Presented) The system of claim 44 wherein the discrepancy 
subsystem is fiirther configured to electronically report the discrepancy to the first evaluator over 
a distributed computer network. 

47. (Previously Presented) The system of claim 44 wherein the discrepancy 
subsystem is further configured to electronically report the discrepancy to a system manager over 
a distributed computer network. 

48. (Previously Presented) A system for electronically providing performance 
feedback based upon a comparison of an evaluator's performance in evaluating data items to 
predetermined performance criteria, the system comprising: 

a) a receiving subsystem configured to electronically receive information over a 
distributed computer network related to a first reliability level of a first evaluator's evaluation of 
data items, and a predetermined reUability level; 

b) a comparison subsystem configured to electronically compare the first reliability level 
and the predetermined reliability level; and 

c) a discrepancy subsystem configured to electronically report over a distributed 
computer network a discrepancy based on predefined criteria between the first reliability level 
and the predetermined reliability level. 
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49. (Previously Presented) The system of claim 48 wherein the receiving 
subsystem is further configured to receive a first evaluation result corresponding to the first 
evaluator's evaluating of a data item during a first time period and to receive a second evaluation 
result corresponding to the first evaluator's evaluation of the data item during a second time 
period. 

50. (Previously Presented) The system of claim 48 wherein the discrepancy 
subsystem is fiirther configured to electronically report the discrepancy to the first evaluator over 
a distributed computer network. 

51 . (Previously Presented) The system of claim 48 wherein the discrepancy 
subsystem is fiirther configured to electronically report the discrepancy to a system manager over 
a distributed computer network. 

52. (Previously Presented) A system for electronically providing performance 
feedback based upon a comparison of an evaluator's performance in evaluating data items to 
predetermined performance criteria, the system comprising: 

a) a receiving subsystem configured to electronically receive information over a 
distributed computer network related to a first speed level of a first evaluator's evaluating of data 
items, and a predetermined speed level; 

b) a comparison subsystem configured to electronically compare the first speed level and 
the predetermined speed level; and 

c) a discrepancy subsystem configured to electronically report a discrepancy over a 
distributed computer network based on predefined criteria between the first speed level and the 
predetermined speed level. 

53. (Previously Presented) The system of claim 52 wherein the receiving subsystem is 
fiirther configured to receive at least one of the following: a first evaluating rate corresponding to 
the first evaluator's evaluating of the data items, an average evaluating rate corresponding to a 
selected group of evaluators' evaluating of the data items, and a second evaluating rate 
corresponding to a second evaluator's evaluating of the data items. 



5 



• AppL No. 10/690,488 
Amdt. dated July 29, 2004 

54. (Previously Presented) The system of claim 53 wherein the discrepancy 
subsystem is further configured to electronically report the discrepancy to the first evaluator over 
a distributed computer network. 

55. (Previously Presented) The system of claim 53 wherein the discrepancy 
subsystem is fiirther configured to electronically report the discrepancy to a system manager over 
a distributed computer network, 

56. (Previously Presented) A system for increasing evaluator efficiency by 
monitoring evaluator performance in evaluating data items and recommending break times to the 
evaluators, the system comprising: 

a) receiving subsystem configured to electronically receive first performance levels over 
a distributed computer network corresponding to an evaluator's evaluating of data items at 
predetermined past intervals in time, and electronically receiving over a distributed computer 
network a second predetermined performance level; 

b) a comparison subsystem configured to electronically compare the second 
predetermined performance level and the first performance level; and 

c) a discrepancy subsystem configured to electronically detect a discrepancy based on 
predefined criteria between the second predetermined performance level and the first 
performance level and signaling the evaluator to hak data item evaluating for a break time. 

57. (Previously Presented) The system of claim 56 wherein the receiving subsystem is 
fiirther configured to electronically receive first validity levels fi'om the evaluator over a 
distributed computer network and to electronically receive a second predetermined validity level 
over a distributed computer network. 

58. (Previously Presented) The system of claim 56 wherein the receiving subsystem is 
fiirther configured to electronically receive first reliability levels fi-om the evaluator over a 
distributed computer network and to electronically receive a second predetermined reliability 
level over a distributed computer network. 

59. (Previously Presented) The system of claim 56 wherein the subsystem is fiulher 
configured to electronically receive first speed levels from the evaluator over a distributed 



6 



• AppL No. 10/690,488 
Amdt. dated July 29, 2004 

computer network and to electronically receive a second predetermined speed level over a 
distributed computer network. 

60. (Previously Presented) The system of claim 56 wherein the discrepancy 
subsystem is further configured to electronically detect a discrepancy based on predefined 
criteria between the second predetermined performance level and the first performance levels 
and providing the evaluator with diversionary activities over a distributed computer network. 

6 1 . (Previously Presented) The system of claim 56 wherein the discrepancy 
subsystem is further configured to enable the evaluator to select diversionary activities over a 
distributed computer network when the evaluator is signaled for the break time. 

62. (Previously Presented) A system for increasing evaluator efficiency by 
monitoring evaluator performance in evaluating data items and recommending break times to the 
evaluator, the system comprising: 

a) a receiving subsystem configured to electronically receive first evaluation rates 
corresponding to an evaluator's evaluating of data items at predetermined past intervals in time, 
and electronically receive a second present evaluation rate corresponding to the evaluator's 
evaluating of the data items at a present time; 

b) a comparison subsystem configured to electronically compare the second evaluation 
rate and the first evaluation rate; and 

c) a discrepancy subsystem configured to electronically detect a discrepancy based on 
predefined criteria between the second evaluation rate and the first evaluation rate as indicated 
by the comparison subsystem and signal over the distributed computer network the evaluator to 
halt data item evaluation for a break time. 

63. (Previously Presented) The system of claim 62 wherein the discrepancy 
subsystem is further configured to electronically detect a discrepancy based on predefined 
criteria between the second present evaluation rate and the first evaluation rate as indicated by 
the comparison subsystem and provide the evaluator with diversionary activities over the 
distributed computer network. 
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64. (Previously Presented) The system of claim 62 wherein the discrepancy 
subsystem is further configured to enable the evaluator to select diversionary activities over the 
distributed network when the evaluator has been signaled for the break time. 

65. (Previously Presented) A computer-readable program storage medium tangibly 
embodying a data package and associated verification instructions executable by a computing 
system for electronically providing performance feedback based upon a comparison of a-an 
evaluator's performance in evaluating data items to predetermined performance criteria, the 
method comprising the steps of: 

a) electronically receiving information related to a first performance level of a first 
evaluator's evaluation of data items, and a predetermined performance level; 

b) electronically comparing the first performance level and the predetermined 
performance level; and 

c) electronically reporting a discrepancy based on predefined criteria between the first 
performance level and the predetermined performance level. 

66. (Previously Presented) The computer-readable program storage medium of claim 
65 wherein the step of electronically reporting a discrepancy comprises electronically reporting 
the discrepancy to the first evaluator. 

67. (Previously Presented) The computer-readable program storage medium of claim 
65 wherein the step of electronically reporting a discrepancy comprises electronically reporting 
the discrepancy to a system manager. 

68. (Previously Presented) A computer-readable program storage medium tangibly 
embodying a data package and associated verification instructions executable by a computing 
system for electronically providing performance feedback based upon a comparison of an 
evaluator's performance in scoring data items to predetermined performance criteria, the method 
comprising the steps of: 

a) electronically receiving information related to a first validity level of a first evaluator's 
evaluation of data items, and a predetermined validity level; 
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b) electronically comparing the first validity level and the predetermined validity level; 

and 

c) electronically reporting a discrepancy based on predefined criteria between the first 
validity level and the predetermined validity level. 

69. (Previously Presented) The computer-readable program storage medium of claim 

68 wherein the step of electronically receiving information comprises receiving an evaluation 
result for a quality item fi'om the first evaluator and a known score for the quality item. 

70. (Previously Presented) The computer-readable program storage medium of 
claim 68 wherein the step of electronically reporting a discrepancy comprises electronically 
reporting the discrepancy to the first evaluator. 

71 . (Previously Presented) The computer-readable program storage medium of claim 
68 wherein the step of electronically reporting a discrepancy comprises electronically reporting 
the discrepancy to a system manager. 

72. (Previously Presented) A computer-readable program storage medium tangibly 
embodying a data package and associated verification instructions executable by a computing 
system for electronically providing performance feedback based upon a comparison of an 
evaluator's performance in evaluating data items to predetermined performance criteria, the 
method comprising the steps of: 

a) electronically receiving information related to a first reliability level of a first 
evaluator's evaluation of data items, and a predetermined reliability level; 

b) electronically comparing the first reliability level and the predetermined reliability 
level; and 

c) electronically reporting a discrepancy based on predefined criteria between the first 
reliabiUty level and the predetermined reliability level. 

73. (Previously Presented) The computer-readable program storage medium of 
claim 72 wherein the step of electronically receiving information comprises receiving a first 
evaluation result corresponding to the first evaluator's evaluation of a data item during a first 
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time period and for receiving a second evaluation result corresponding to the first evaluator's 
evaluation of the data item during a second time period. 

74. (Previously Presented) The computer-readable program storage medium of 
claim 72 wherein the step of electronically reporting a discrepancy comprises electronically 
reporting the discrepancy to the first evaluator. 

75. (Previously Presented) The computer-readable program storage medium of 
claim 72 wherein the step of electronically reporting a discrepancy comprises electronically 
reporting the discrepancy to a system manager. 

76. (Previously Presented) A computer-readable program storage mediimi 
tangibly embodying a data package and associated verification instructions executable by a 
computing system for electronically providing performance feedback based upon a comparison 
of an evaluator's performance in evaluation of data items to predetermined performance criteria, 
the method comprising the steps of: 

a) electronically receiving information related to a first speed level of a first evaluator's 
evaluation of data items, and a predetermined speed level; 

b) electronically comparing the first speed level and the predetermined speed level; and 

c) electronically reporting a discrepancy based on predefined criteria between the first 
speed level and the predetermined speed level. 

77. (Previously Presented) The computer-readable program storage mediimi of 
claim 76 wherein the step of electronically receiving information comprises receiving at least 
one of the following: a first evaluation rate corresponding to the first evaluator's evaluation of the 
data items, an average evaluation rate corresponding to a selected group of evaluators' evaluation 
of the data items, and a second evaluation rate corresponding to a second evaluator's evaluation 
of the data items. 

78. (Previously Presented) The computer-readable program storage medium of 
claim 76 wherein the step of electronically reporting a discrepancy comprises electronically 
reporting the discrepancy to the first evaluator. 
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79. (Previously Presented) The computer-readable program storage medium of 
claim 76 wherein the step of electronically reporting a discrepancy comprises electronically 
reporting the discrepancy to a system manager. 

80. (Previously Presented) A computer-readable program storage medium 
tangibly embodying a data package and associated verification instructions executable by a 
computing system for increasing evaluator efficiency by monitoring evaluator performance in 
evaluating data items and recommending break times to the evaluator, the method comprising the 
steps of: 

a) electronically receiving first performance levels corresponding to an evaluator's 
evaluation of data items at predetermined past intervals in time, and electronically receiving a 
second predetermined performance level; 

b) electronically comparing the second predetermined performance level and the first 
performance level; and 

c) electronically detecting a discrepancy based on predefined criteria between the second 
predetermined performance level and the first performance level and signaling the evaluator to 
halt data item evaluation for a break time. 

8 1 . (Previously Presented) The computer-readable program storage medium of 
claim 80 wherein the step of electronically receiving first performance levels comprises 
electronically receiving first validity levels firom the evaluator and electronically receiving a 
second predetermined validity level. 

82. (Previously Presented) The computer-readable program storage mediiun of 
claim 80 wherein the step of electronically receiving first performance levels comprises 
electronically receiving first reliability levels from the evaluator and electronically receiving a 
second predetermined reliability level. 

83. (Previously Presented) The computer-readable program storage medium of claim 
80 wherein the step of electronically receiving first performance levels comprises electronically 
receiving first speed levels from the evaluator and electronically receiving a second 
predetermined speed level. 
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84. (Previously Presented) The computer-readable program storage medium of claim 
80 wherein the step of electronically detecting a discrepancy comprises electronically detecting a 
discrepancy based on predefined criteria between the second predetermined performance level 
and the first performance levels and providing the evaluator with diversionary activities. 

85. (Previously Presented) The computer-readable program storage medium of claim 
80 wherein the step of electronically detecting a discrepancy comprises enabling the evaluator to 
select diversionary activities when the evaluator is signaled for the break time. 

86. (Previously Presented) A computer-readable program storage medium tangibly 
embodying a data package and associated verification instructions executable by a computing 
system for increasing evaluator efficiency by monitoring evaluator performance in evaluating 
data items and recommending break times to the evaluators, the method comprising the steps of: 

a) electronically receiving a first evaluation rate corresponding to an evaluator's 
evaluating of data items at predetermined past intervals in time, and electronically receiving a 
second present evaluation rate corresponding to the evaluator's scoring of the data items at a 
present time; 

b) electronically comparing the second evaluation jate and the first evaluation rate; and 

c) electronically detecting a discrepancy based on predefined criteria between the second 
evaluation rate and the first evaluation rate as indicated by the comparison step and signaling the 
evaluator to halt data item evaluation for a break time. 

87. (Previously Presented) The computer-readable program storage medium of claim 
86 wherein the step of electronically detecting a discrepancy fiirther comprises electronically 
detecting a discrepancy based on predefined criteria between the second present evaluation rate 
and the first evaluation rate as indicated by the comparison step and providing the evaluator with 
diversionary activities. 

88. (Previously Presented) The computer-readable program storage medium of claim 
86 wherein the step of electronically detecting a discrepancy fiirther comprises enabling the 
evaluator to select diversionary activities when the evaluator has been signaled for the break 
time. 
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89. (Previously Presented) A method for electronically providing performance 
feedback based upon a comparison of an evaluator's performance in evaluating data items to 
predetermined performance criteria, the method comprising the steps of: 

a) electronically receiving information related to a first performance level of a first 
evaluator's evaluation of data items, and a predetermined performance level; 

b) electronically comparing the first performance level and the predetermined 
performance level; and 

c) electronically reporting a discrepancy based on predefined criteria between the first 
performance level and the predetermined performance level. 

90. (Previously Presented) The method of claim 89 wherein the step of electronically 
reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator. 

91 . (Previously Presented) The method of claim 89 wherein the step of electronically 
reporting a discrepancy comprises electronically reporting the discrepancy to a system manager. 

92. (Previously Presented) A method for electronically providing performance 
feedback based upon a comparison of an evaluator's performance in scoring data items to 
predetermined performance criteria, the method comprising the steps of: 

a) electronically receiving information related to a first validity level of a first evaluator's 
evaluation of data items, and a predetermined validity level; 

b) electronically comparing the first validity level and the predetermined validity level; 

and 

c) electronically reporting a discrepancy based on predefined criteria between the first 
validity level and the predetermined validity level. 

93. (Previously Presented) The method of claim 92 wherein the step of electronically 
receiving information comprises receiving an evaluation result for a quality item from the first 
evaluator and a known score for the quality item. 

94. (Previously Presented) The method of claim 92 wherein the step of electronically 
reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator. 
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95. (Previously Presented) The method of claim 92 wherein the step of electronically 
reporting a discrepancy comprises electronically reporting the discrepancy to a system manager. 

96. (Previously Presented) A method for electronically providing performance 
feedback based upon a comparison of an evaluator*s performance in evaluating data items to 
predetermined performance criteria, the method comprising the steps of: 

a) electronically receiving information related to a first rehability level of a first 
evaluator^s evaluation of data items, and a predetermined reliability level; 

b) electronically comparing the first reliability level and the predetermined reliability 
level; and 

c) electronically reporting a discrepancy based on predefined criteria between the first 
reliability level and the predetermined reliability level. 

97. (Previously Presented) The method of claim 96 wherein the step of electronically 
receiving information comprises receiving a first evaluation result corresponding to the first 
evaluator's evaluation of a data item during a first time period and for receiving a second 
evaluation result corresponding to the first evaluator's evaluation of the data item during a 
second time period. 

98. (Previously Presented) The method of claim 96 wherein the step of 
electronically reporting a discrepancy comprises electronically reporting the discrepancy to the 
first evaluator. 

99. (Previously Presented) The method of claim 96 wherein the step of 
electronically reporting a discrepancy comprises electronically reporting the discrepancy to a 
system manager. 

100. (Previously Presented) A method for electronically providing performance 
feedback based upon a comparison of an evaluator's performance in evaluation of data items to 
predetermined performance criteria, the method comprising the steps of: 

a) electronically receiving information related to a first speed level of a first evaluator's 
evaluation of data items, and a predetermined speed level; 

b) electronically comparing the first speed level and the predetermined speed level; and 
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c) electronically reporting a discrepancy based on predefined criteria between the first 
speed level and the predetermined speed level. 

101. (Previously Presented) The method of claim 1 00 wherein the step of 
electronically receiving information comprises receiving at least one of the following: a first 
evaluation rate corresponding to the first evaluator's evaluation of the data items, an average 
evaluation rate corresponding to a selected group of evaluators* evaluation of the data items, and 
a second evaluation rate corresponding to a second evaluator's evaluation of the data items. 

102. (Previously Presented) The method of claim 100 wherein the step of 
electronically reporting a discrepancy comprises electronically reporting the discrepancy to the 
first evaluator. 

103. (Previously Presented) The method of claim 100 wherein the step of 
electronically reporting a discrepancy comprises electronically reporting the discrepancy to a 
system manager. 

104. (Previously Presented) A method for increasing evaluator efficiency by 
monitoring evaluator performance in evaluating data items and recommending break times to the 
evaluators, the method comprising the steps of: 

a) electronically receiving first performance levels corresponding to an evaluator's 
evaluation of data items at predetermined past intervals in time, and electronically receiving a 
second predetermined performance level; 

b) electronically comparing the second predetermined performance level and the first 
performance level; and 

c) electronically detecting a discrepancy based on predefined criteria between the second 
predetermined performance level and the first performance level and signaling the evaluator to 
halt data item evaluation for a break time. 

105. (Previously Presented) The method of claim 104 wherein the step of 
electronically receiving first performance levels comprises electronically receiving first validity 
levels fi"om the evaluator and electronically receiving a second predetermined validity level. 
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106. (Previously Presented) The method of claim 104 wherein the step of 
electronically receiving first performance levels comprises electronically receiving first 
reliability levels from the evaluator and electronically receiving a second predetermined 
reliability level. 

107. (Previously Presented) The method of claim 104 wherein the step of 
electronically receiving first performance levels comprises electronically receiving first speed 
levels from the evaluator and electronically receiving a second predetermined speed level. 

108. (Previously Presented) The method of claim 104 wherein the step of 
electronically detecting a discrepancy comprises electronically detecting a discrepancy based on 
predefined criteria between the second predetermined performance level and the first 
performance levels and providing the evaluator with diversionary activities. 

109. (Previously Presented) The method of claim 104 wherein the step of 
electronically detecting a discrepancy comprises enabling the evaluator to select diversionary 
activities when the evaluator is signaled for the break time. 

110. (Previously Presented) A method for increasing evaluator efficiency by 
monitoring evaluator performance in evaluating data items and recommending break times to the 
evaluators, the method comprising the steps of: 

a) electronically receiving a first evaluation rate corresponding to an evaluator's 
evaluating of data items at predetermined past intervals in time, and electronically receiving a 
second present evaluation rate corresponding to the evaluator's scoring of the data items at a 
present time; 

b) electronically comparing the second evaluation rate and the first evaluation rate; and 

c) electronically detecting a discrepancy based on predefined criteria between the second 
evaluation rate and the first evaluation rate as indicated by the comparison step and signaling the 
evaluator to halt data item evaluation for a break time. 

111. (Previously Presented) The method of claim 110 wherein the step of 
electronically detecting a discrepancy further comprises electronically detecting a discrepancy 
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based on predefined criteria between the second present evaluation rate and the first evaluation 
rate as indicated by the comparison step and providing the evaluator with diversionary activities. 

112. (Previously Presented) The method of claim 110 wherein the step of 
electronically detecting a discrepancy further comprises enabling the evaluator to select 
diversionary activities when the evaluator has been signaled for the break time. 
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